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The Academic Pathways of People Learning Engineering Survey (APPLES or APPLE survey) is 
a component of the Academic Pathways Study (APS) of the Center for the Advancement of 
Engineering Education (CAEE). The APS aims to provide a comprehensive account of how 
people become engineers by exploring key questions around the engineering learning experience. 
The goal of the APPLE survey is to validate the longitudinal data from the APS Persistence in 
Engineering (PIE) survey, as well as findings from other 
APS data collection methods. 


Lessons Learned 

Despite the Team’s experience with the PIE survey 
deployment during 2004-2007 and comprehensive 
planning, many lessons were learned in the design and 
administration of the APPLE survey: 


• Start the Internal Review Board (IRB) process 
at least four months prior to the planned deployment. This is especially important for 
multi-site studies where multiple IRB approvals are required. 

• Consider the different stakeholders when determining the incentive for 
participation. In addition to the potential student respondents, the team recognized that 
incentives or other compensation may be needed for institutions’ and their liaisons’ 
participation. 

• Allocate resources for comprehensive piloting of the survey. Measuring the time-to- 
take of the survey instrument during piloting was particularly valuable in guiding difficult 
decisions about which items and constructs to keep or delete. 

• Emphasize sharing and learning from other recruitment plans across institutions. 
Recruitment plans were shared across institutions with limited success because liaisons 
reported that the framework was helpful in developing their own campus recruitment 
plans, but they did not fully appreciate the other institutions’ plans until after the 
deployment. 

• Add survey items to weed out ineligible subjects. The team used responses in the 
demographic questions to easily identify ineligible subjects (such as graduate students 
and students from other universities) and eliminate them from the analyses. 

• Allow multiple submissions from a single IP address. The decision to allow more than 
one submission per IP address proved to be a good one as 22% of one school’s 
submissions came from a single IP address, presumably due to the firewall system used 
there. 


...in order to validate longitudinal 
data in one survey administration, 
the target sample needed to be 
cross-sectional and include 
respondents from all stages of the 
undergraduate academic degree. 


Apple Survey Deployment and Recruitment of Participants 

The APPLE survey will be deployed twice (APPLES 1 and APPLES2), to two different 
populations of American undergraduate engineering students. APPLES 1 was administered in 
April 2007 and targeted students from the same four schools as APS' Cohort 1 (Coleman 
University, Mountain Technical Institute, Oliver University, and University of West State; 
pseudonyms). 

Due to APS’ focus on persistence in engineering education, APPLES recruitment targeted three 
groups of undergraduate students: (1) engineering students: those who have declared an 
engineering major or have already committed to engineering programs; (2) pre-engineering 
students: those who intend to declare an engineering major; and (3) non-persister students: those 
who were initially interested in majoring in engineering, but have since decided to pursue a non- 
engineering major. 

The engineering program structures at the four APS institutions differ significantly. Recruitment 
methods that are successful at one institution may not be appropriate or even possible at another 
institution. Thus, the team was under certain constraints in devising a sampling plan. The team 
had to use a convenience sampling method whereby targeted respondents were requested to 
voluntarily take the APPLE survey as compared to a random sampling where the team could 
choose the subjects randomly. Also, in order to validate longitudinal data in one survey 
administration, the target sample needed to be cross-sectional and include respondents from all 
stages of the undergraduate academic degree (see full paper at the link below for recruitment 
method details). 

In designing the APPLES administrations, the APPLES 1 deployment was intended to be a model 
for the larger-scale APPLES2. Campus liaisons worked with the team at each of the four 
institutions. These campus liaisons assisted in securing IRB human subjects approvals, planning 
campus-specific recruitment, and coordinating local deployment. The liaisons also provided 
valuable insights into their institutional and student cultures, campus infrastructure, and student 
preferences during the development of recruitment materials and plans. None of the institutions 
reached their targets for all strata (pre-sampling grouping of the target population, e.g., 
persistence and academic class). Deployment dates at various institutions were constrained by 
spring break schedules, final administration of the Cohort 1 PIE survey, the end of the academic 
year at semester institutions, and response rates. 

IRB approval was required for each of the four institutions because all had researchers who were 
involved in the APS. Universal IRB requirements, such as not requiring the subject to answer 
one or any of the survey questions, significantly guided the PIE and APPLE survey designs. 
Institution-specific IRB requirements did vary. 

Information consultations with other survey researchers suggested that recruiting engineering 
undergraduate students was more difficult than recruiting other students and that they should be 
offered an incentive for participation. A $4 electronic payment was chosen and offered to every 
individual who took the survey through a commonly-used financial transaction company. 

Patterns of responses from the Cohort 1 respondents, who the team believes to perceive the 
survey with moderately high levels of importance and interest, encouraged the team to expand 


CAEE Research Brief 


2 



recruitment efforts to include basic recruitment methods aimed at a larger audience: poster, 
emails to student distribution lists, and an ad in the student newspaper - all of which carried 
consistent branding of a red apple logo. The IRBs required the survey be open to all students. 
However, posters were designed to focus on the target respondents: pre-engineering, 
engineering, and non-persister students. Because the four institutions differed in terms of student 
demographics and when students were required to declare their majors, recruitment targets were 
set on a per school basis and campus liaisons received daily reports on responses for each 
recruitment strata to help direct recruitment efforts. 

The APPLE survey underwent two rounds of piloting: a first round primarily to refine the survey 
questions for clarity and identify questions for elimination, and a second to streamline the final 
product. The length of the survey was a primary concern given the limited compensation of 
respondents. Using timing data that recorded how long respondents spent answering individual 
survey questions during the first pilot, the team was able to pare down the survey instrument to 
the target time of approximately 10 minutes. 

There were over 900 submissions for the first APPLES administration. The average response rate 
was 17 percent across the four institutions. APPLES2 will be deployed in early 2008 at 22 
institutions in the US. 
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